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Abstract 



We model and study the genetic evolution and conservation of a population of diploid 
hermaphroditic organisms, evolving continuously in time and subject to resource corn- 
ed ■ petition. In the absence of mutations, the population follows a 3-type nonlinear 
birth-and-death process, in which birth rates are designed to integrate Mendelian 
reproduction. We are interested in the long term genetic behaviour of the popula- 
tion (adaptive dynamics), and in particular we compute the fixation probability of 
^ ■ a slightly non-neutral allele in the absence of mutations, which involves finding the 
I unique sub-polynomial solution of a nonlinear 3-dimensional recurrence relationship. 
04 ■ This equation is simplified to a 1-order relationship which is proved to admit exactly 
one bounded solution. Adding rare mutations and rescaling time, we study the succes- 
sive mutation fixations in the population, which are given by the jumps of a limiting 



I Markov process on the genotypes space. At this time scale, we prove that the fixation 



rate of deleterious mutations increases with the number of already fixed mutations, 
which creates a vicious circle called the extinction vortex. 



Keywords: Population genetics, diploid population, nonlinear birth-and-death process, fixation 
■ probability, Dirichlet problem, multidimensional nonlinear recurrence equations, extinction vortex. 



1 Introduction 

Our goal is to model a finite population with diploid reproduction and competition. We 
specially want to understand the role of diploidy and Mendelian reproduction on mutation 
fixation probabilities and on the genetic evolution of a population. We are interested in 
studying the progressive accumulation of sm all deleterious mu t ation s which generates an 



extinc tion vortex in sni all populations (see iGilpin and Soulel (|1986| ): iLvnch and Gabriel 



( 1990l ) and Coron et al. for more biological context and analyses). 
The population follows a birth-and-death process in which each individual has a natural 
death rate that depends on its genotype (Section |2]). Birth rates are designed to model 
the Mendelian reproduction, and individuals are competing against each other. First, 
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in the absence of mutation, we focus on one gene and compute the fixation probabil- 
ity of an allele a competing aga inst a resident allele A (Sections |3] and |4| as done in 
Champagnat and LambertI (120071 ) for the simpler haploid case. We first consider the neu- 
tral case, where individuals all have same birth, natural death and competition death rates 
(i.e. alleles A and a are exchangeable). Here a martingale argument proves that the fix- 
ation probability of allele a is simply equal to the initial proportion of this allele in the 
population. We next consider the case where allele a is slightly non-neutral, i.e. natural 
death rates slightly deviate from the neutral case. Here we prove that the fixation proba- 
bility of allele a is differentiable in the parameters of deviation from the neutral case and 
that its partial derivatives are the unique subpolynomial solutions of Dirichlet problems. 
These equations consist in 3-dimensional nonlinear double recurrence relationships which 
we manage to simplify to a 1-dimensional double recurrence admitting a unique bounded 
solution. In Section |5l we add rare mutations and rescale time in order to observe mu- 
tation apparitions. At this time scale, mutations get fixed or disappear instantaneously, 
and the successive fixations of mutations are given by the jumps of a Markov p rocess S on 



the genotypes space, called th e "Trait Subs t itutio n Se quence", introduced by iMetz et al. 



(|l996) and studied notably in IChampagnatI (j2006l ) and ICoUet et al.l (120121 ) in the diploid 
case. Here the population size remains finite, and we do not use any deterministic ap- 
proximation. We finally get interested in the successive jump rates of S in the particular 
case of deleterious mutations (Section 15. 3p . Indeed we prove that when every mutation 
is deleterious, the Markov process S jumps more and more rapidly, i.e. the fixation rate 
of a deleterious mutation increases with the number of already fixed mutations, if the 
population i s small enough which creates a vicious circle called the extinction vortex (see 
Coron et al.l for biological interpretations and numerical results). 



2 Presentation of the model 



We consider a population of diploid hermaphrod itic self-incompatib l e orga n isms, charac- 



terized by their genotypes. Bu ilding on works of IChampagnat et al.l (120061 ) ; IChampagnat 



(|2006l ) and ICoUet et al.l (120121 ) . we consider a birth and death process with mutation, se- 
lection and competition under different time scales and we add diploidy. Each individual 
is characterized by its genotype x € G := {{A,C,Q,T}^}^ where G is the genome size 
and A, C, Q, and T are the four nucleotides that compose DNA. Genotype x = {xi,X2) is 
in fact composed with two DNA strands xi and X2 in {A,C,Q,T}'~' ■ In Sections |2] to IH we 
consider the case without mutation and assume that the population is initially composed 
with individuals that only differ from each other on one gene. For this gene, there are two 
possible alleles, denoted by A and a in {A,C,Q,T}^ where G' ^ G. The genotypes of 
individuals are thus denoted AA, Aa, and aa, and we represent the population dynamics 
by the Markov process: 

Z -.tt-^ Zt = {kt,mt,nt), 

that gives the respective numbers of individuals with genotype AA, Aa, and aa at time t. 
For more simplicity, we will also refer to these genotypes as types 1, 2, and 3. We assume 
that the process Z is a birth-and-death process with competition on N^, and we now detail 
the birth and death rates of individuals of each genotype. The population has maximum 
fecundity rate r. More precisely, if the population contains N individuals, rN is the rate at 
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which two distinct individuals of the population encounter, and the maximum total birth 
rate. These two individuals are chosen uniformly randomly in the population, and their 
encounter gives rise to a birth with a probability pij [pij = pji) that depends on their two 
genotypes i and j. pij can be defined biologically as the selective value associated with 
the couple of genotypes i and j, and represents both the degree of adaptation of types i 
and j and their compatibility. Finally the new-born individual results from a segregation 
(genetic melting between the genotypes of its parents), satisfying Mendel's laws of heredity. 
Then in the population Z = (k, m, n) such that A; + m + n ^ 2, if we define hij := rpij^ the 
rate bi[Z) at which an individual of type i G {1,2,3} arises is: 

, fr^\ 1 k(k - 1) , km , m(m - 1) 
6i(Z) = 5ii-^- — -l + bu— — 7 + 622- 



N-1 4(Af-l)' 

, , km , m(m — 1) , mn , 2kn , , 

h2{Z) = hyi Vh22—7 r + 623 Vhxz , (1) 

, , nin — 1) , mn , mim — 1) 

b.(Z) = b33— + 623 h 622^^^ r- 

^ N-1 ^ N-1 4{N-1) 

Note that if the population Z has size N, 

bi{Z) + b2iZ) + b3{Z) i^rN. (2) 

We assume self-incompatibility, which implies that when the population size reaches 1, no 
birth can occur anymore and the population can be considered as extinct. Now individuals 
can die either naturally or due to competition with others. We denote by di the natural 
death rate of individuals with type i and Cij the competition rate of i against j, i.e. the 
rate at which a fixed individual of type i makes a fixed individual of type j die. We assume 

Cij > G {1,2,3}, i.e. c= inf Cij > (3) 

i,iG{l,2,3} 

and that when the population size reaches 2, no death can occur, hence the population 
cannot get extinct. We then denote the state space of Z by 

= N3 \ {(0, 0, 0), (1,0, 0), (0, 1, 0), (0, 0, 1)}. 

In the population Z = {k, m, n) such that k + m + n ^ 3, the rate d^^^ {Z) at which the 
population loses any individual of type i then is: 

(Z) = (di + cii {k -1) + C2im + C3in)k, 
d^"^^ (Z) = {d2 + ci2k + C22{m - 1) + C32n)m, (4) 
d(^) (Z) = {ds + cisk + C23"i + C33(n - l))n, 

and if A; + m + n = 2, 

d(^) (Z) = (z) = (Z) = 0. (5) 

From ([2]), dSl, and Theorem 2.7.1 in iNorrisI (Il997t ). the process Z does not explode. Then 
Zt is defined for all t > 0, and we denote by V(k,m,n) the law of Z starting from state 
{k,m,n), E(-;j, ,„,^) the associated expectation, {Zi)i^fq the embedded Markov chain, and 
(J7)/gis} the filtration generated by Z. 
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Notation: For every other process X, is the law of X starting from Xq, and 

is the associated expectation. If X is a continuous-time (resp. discrete time) process, we 

denote (resp. T^) the reaching time of x by X. 

In the following, the population size process will play a main role; we define N : t Nt = 
[kt + 711,^-1- rit) where = (kt,mf,nf), for every time t > and {J\fi)i^^ the embedded 
Markov chain. is stochastically dominated by the logistic birth-and-death process Y 
with transition rates: 

rj if j = i + 1, 

cj{j - 1) if J = z - 1 and i ^ 2, (6) 
otherwise 

We define y the embedded Markov chain. 

Proposition 2.1. For all N G'N, there exists p > such that EAr((l + p)'^^) < oo. 

Proof. Let Nq be such that b < (d + c{Nq — 1)). We assume that N > Nq, without loss 
of generality. Note that it suffices to prove that for every integer n € [3, A^], there exists 
Pn > such that 

En((l + /9„)^"-i) <00. (7) 



Indeed , ^^{{l+p)'^^) = H^Ls < oo if p ^ infj pi. Now, from lSeneta and Vere-Jones 



(Il966[ ) p. 428, m s true for n = N, since N > Nq. Now, following the proof of Lemma 



,^f lCollet et all fc^, let us prove by mduction that if © is true forn+ 1 then 



it is also true for n. We assume that ([7} is true for n + 1 and that Yq = n, and we define 
M the random number of returns in n before going to n — 1. M follows a geometrical law 
with parameter p = b/{b + d + c{n — 1)). Then 



M 



1=1 



where the Tn^i are independent and distributed as 7^ for all i. Then by strong Markov 
Property in the stopping times Tn,!, we obtain 

oo 
m=0 

Finally, since ([7| is true for n+l, from the Dominated Convergence Theorem, E„+i ^(1 + 

goes to 1 when p goes to 0, hence there exists pn-i such that E„+i ((1 + pn-if"+^) < l/p 
which gives the result. □ 



Proposition 2.2. For all p ^ 1, ifE{N^) < oo then supE(iVf) < oo. 
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Proof. We set Yq = Nq. It suffices to prove that sup^E(l^^) < oo. (yi)t>o is a recurrent, 
irreducible, and ergodic Markov process on N \ {0, 1}, with stationary law / (see Equation 

oo 

(|38p for a more general case), and we can easily check that Ep := ^^/(i)j^ < oo for all p. 

i=2 

Now let us define the Markov process (Yf, Zt)t^o such that Y and Z have same transition 
rates, are independent, and Zq has law /. We define (3^n)^n)neN the ass ociated Markov 
chain, and T = inf{n|3^n = Z^}- Following the proof of Theorem 6.6.4 in Durrett ( 2010f ) 
p. 308, we have 



my^ - Ep\ = myp) - e{zp)\ ^ y1 ^"i^iyn = ^) - n^n = z)\ 

^ ^ zPi¥{yn = z,r>n)+ P(3^„ = z,T> n)) 

= Ei{yp + zpjir>n) 

^ 2E{yPlr>nlYo>Zo) + 2E{ZPlr>nlZo>Yo) 



Now 



^ 2P ^ zP¥{Yq = z) + 2P nPp(7f ^ n; = z) 
^ 2P ^ zPF{Yo = z) + 2PnPF{Ti' ^ n). 



z^n 



From Proposition [2?T1 and since E(yJ') < oo, nPW{T2 ^ n) and X]^>„ zP¥(Yq = z) converge 
to 0. Then E(l^) converges to Ep when n goes to infinity. Since Y does not explode and 
E(F(f) < oo, we have supE(yj^) < oo. □ 



3 Fixation probabilities 
3.1 Absorbing states 

The birth and death process Z admits the following absorbing states sets: 

• Fa = {(0, 0, n), n ^ 2} is the set of states for which allele a is fixed and allele A has 
disappeared. 

• F^ = {(A;, 0, 0), /c ^ 2} is the set of states for which allele A is fixed and allele a has 
disappeared. 

. F := Fa U F^ 

We are interested in computing the probability that allele a goes to fixation (i.e. Z reaches 
Fa), when Z starts from any state (/c, m, n). We now define 7h the (discrete) reaching time 
of set Q by ^ for all O C NL . Th e following result is an adaptation of Proposition 6.1. in 



Champagnat and LambertI (120071 ) to the diploid case. 
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Proposition 3.1. There exists a constant C such that for any initial state {k,m,n) in 
NL; IE(fc,m,„)(7'r) ^Cik + m + n). 

Proof. Let 7{2} be the first time wliere tlie Markov cliain J\f readies 2 (or returns to 2 if 
A/q = 2), and define 

T{2}->r := sup ^(k,m,n){Tr)- 

(k,m,n)\ 
k+m+n=2 

Then Ei^k,m,n){Tr) ^_K(k,m,n){T{2}) + T{2}^r, and T{2}^r is independent of {k,m,n). 
We prove first that 7~{2}->.r < oo and second that tliere exists a constant Ci such that 
^{k,m,n){'T{2}) <Ci{k + m + n) for all {k,m,n) in N^^. Now, 



T{2}^r - sup E(fc,„,,) (^7rl{r{2}^rr} + '7rl{r{2}<rr} 



{k,m,n)\ 
fc+m+n=2 

^ sup IE(fc,„,„)(r{2}) + sup Ei^k,m,n)({Tr -T{2})^{Ti2}<Tr}) 
{k,m,n)\ {k,m,n)\ ^ ^ 

fc+m+n=2 fc+m+?i=2 

< sup E(k,m,n)iT{2}) 

(fc,m,n)| (oj 
fc+m+n=2 

+ sup ^ f(7f^ - T[2})l{r{2}<rr}lzT,2, =(fc',m',n')) 

(fc,m,n)| n I I i\i 
' " (k',m',n')\ 
k+m+n=2 ,} ' , ' /' 
k +m +n =2 

< sup IE(fc,m,n)('7{2}) +T{2}^r SUp P(fc,m,n) ({'7{2} < 7r}) , 
(fc,m,n)| (fc,TO,n)| 

A;+m+n=2 fc+TO+n=2 

where the last inequality is obtained using the strong Markov property in T{2}- Defining 

p= sup ^(k,m,n){T{2} <Tt) and 

(k,m,n)\k+m+n=2 

'^{2}^{2} = sup „, „)(7{2}), 

(k,m,n)\k+m+n=2 

we have p < 1, since for every {k,m,n) such that k + m + n = 2, there exists a path 
for Z starting from (k, m, n) and reaching V before reaching the set {A^ = 2}. Besides, 
T{2}^{2} is bounded by the expectation of the mean time of coming back in {A^ = 2} 
for the process Y defined by Equation (j6|). So 7'{2}^{2} < oo, from Theorem 3.3.3 of 



NorrisI (119971 ) . Finally, from (j8|), (1 —p)T{2}^T ^ '7'{2}^{2}; then T{2}->r < oo- Now, 
let us consider the Markov chain (3^n)nGN on N \ {0, 1}, associated with Y . Af being 
stochastically dominated by 3^, if = fc + m + n, IE(A:,m,n)('7{2}) ^ IE^(inf{n|3^„ = 2}). 
Define S^^i = E^(inf{n[3^„ = i}) and let Nq ^ 2 be a natural integer such that ^ |. 

If A^ ^ A'o then Sn,2 = Sn,No + Sno,2- Moreover, since ^ ^ for all A^ ^ A'o, 

Sn,No ^ E(C/Ar^7Vo) where Un,! is the first reaching time of i, for the discrete time random 
walk on Z starting from A^ and having probability 1/3 to jump one step on the right and 
2/3 to iump one step on the left, for every state. We know that ¥.(Un^No) ~ ^(-^ ~ -^o) 



NorrisI (|l997l ). pp. 21 - 22. So if AT ^ Nq, E{Sn,2) ^ HSno,2) + 3(A^ - A^o)- Then there 



exists a constant Ci > such that E,{Sn,2) < CiN for all N ^ 2. □ 
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We now consider the fixation probabilities of allele o as a function of the initial state of 
the population. We define Fa = {{Zt)t>o reaches Fq} and u{Z) = ¥.z{^Fa) is the fixation 
probability of allele a knowing that the population starts from state Z. u also depends 
on the demographic parameters of the population, and this dependence will be explicitely 
written down when necessary. Note that {u{Zt))t>o is a martingale since 

u{Zt) = u{kt,mt,nt) =EzAiFj =H^FjJ^t). (9) 

In the neutral case (Section 13. 2p . a martingale argument gives us the value of u, and in the 
non-neutral case with small mutation assumption (Section 13. 3p . we prove that u admits a 
Taylor expansion in the parameters of deviation from the neutral case. 



3.2 Neutral case 

We now consider the neutral case when ecological parameters do not depend on genotypes, 
i.e. when bij = b, Cij = c, and di = d for all i and j in {1, 2, 3}. We first prove the 

Proposition 3.2. In the neutral case, for all {k,m,n) in N^^ and for all ecological param- 
eters b, d and c, 

. , . m + 2n 

u(k, m, n) 



2{k + m + n) 

Proof. Let us define the function p : (k, m, n) i— )• (m + 2n)/2{k + m + n) and denote by T; 
the l-ih jump time of the population (i.e. the time at which occurs the i-th event, birth 
or death). The Markov chain {p{Zi))i(zm gives the successive proportions of allele a in the 
population. We now prove that p(-E/))/£pj is a J^-bounded martingale. To this aim, we 
distinguish two types of states: those where the population size is greater or equal to 3 
and those where it is equal to 2. For Zi = {ki,mi,ni) such that A// ^ 3, one can compute 
'E{p{Zi^i)\Zi) by decomposing it according to the nature of the / + 1-th event: 

Eip{Zi+,)\Zi) = ^-^^^f ~^ P(&at/. of aa) + ^^^M^^Lllp(deat/i of Aa) 

2AMZz)„., ,, ^ , , 2Mip{Zi) + 2 ^^^. 
'2N\^^ ^ — 2A/7+2 — ^^^'^^^^ 

2Mp(^,) + 1 ^^f^^^Fib^rth of AA) 

2Mi + 2 ^ ■' ' 2Mi + 2 ^ ■' ' 

The same result can be easily proved for Ni = 2. From Doob's stopping time theorem 
applied to the bounded martingale {p{Zi))i and to the stopping time 7r (a-s. finite, from 



Proposition 13. ip . we get: 
Now 

Ek,m,n{Pi^Tr)) = Efc,m,n(p(2:rr)lTr,<Tr^ ) +Ek,m,n{Pi^Tr)'^Tr^>Tr J 

= IPfc,m,n(7ra <Tta) = u{k,m,n) 
since Kk,m,niP(.^Tr)\Tra < Tfa) = 1 and Ek,m,niPi^Tr)\Tra > ^Ta) = 0- ^ 
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When the mutation is not neutral, we do not obtain any closed formula for p{Z) as previ- 
ously. We instead consider the Dirichlet problem satisfied by u. 



3.3 Deviation from the neutral case 
3.3.1 A Dirichlet Problem 

We now arbitrarily assume that allele a is slightly deleterious, i.e. the demographic pa- 
rameters {bij)ij, {cij)ij, and are less advantageous for genotypes Aa and aa than for 
genotypes AA, and slightly deviate from the neutral case. This l atter assumption (small 
mutation sizes) is justified in biology papers such as [Or3 (Il998l . Il999l ) which show that 



species evolution is partly due to the fixation of a large number of small mutations. Be- 
sides, we assume that carrying allele a only influences the natural death rate of individuals. 
More precisely, we set 

bij=b 

Cij = c, whereas (10) 

di = d, d2 =d + 5 and d^ = d + 5' , 

where 5 and 5' are close to 0. Note that if 5' is positive and 6 is equal to 0, then allele a 
is deleterious. The effect of 6 is more intricate because it affects heterozygous individuals, 
with the same apparent effect on both alleles. It simply represents a more or less impor- 
tant adaptation of heterozygotes compared to AA homozygotes and as we will see later 
(Subsection 13.3.2]) . its role in the deleterious or positive effect of allele a depends on the 
initial genetic repartition of the population. We denote by L^'^ the infinitesimal generator 
of Z with assumptions (llOp . and by u{{k,m,n),6,6') the fixation probability of allele a, 
knowing that Z starts from {k,m,n), for all {k,m,n) in N^^. We then have for all real 
bounded function / on N^^: 

(L^'^7)(^, m, n) = h{Z)f{k + 1, m, n) + b2{Z)f{k, m + 1, n) + b^{Z)f{k, m, n + 1) 

+ d^^\z)f{k - l,m,n) + d^^\z)f{k,m-l,n) + d^^\z)f{k,m,n-l) 
- {bN +{d + c{N - 1))N + 5m + S'n)f{k, m, n). 

We define from ([U, (|4]), and ([5]), the infinitesimal generator 
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{Lv){k,m,n) = {L^'^v){k,m,n) 
b 



N-1 



k{k — 1) + km + 



m(m — 1] 



v{k + 1, m, n) 



m(m — 1) 1 \ /, 

+ I km H h mn + 2Kn w (A;, m + l,n) 



I m(m — 1) , 
+ I n{n — 1) + mn H I v{k, m,n + 1) 



{Lv){k, m, n 



+ {d + c{N — 1)) [kv{k — l,m, n) + mv{k, m — l,n) + nv{k, m,n — 1)] 
- {bN + dN + cN{N - l))v{k, m,n) if A; + m + n ^ 3, 
b 



N - 1 



k{k — 1) + km H ) v(k + 1, m, n) 



m{m — 1) , 1 

+ I km H h mn + 2kn I v{k, m + l,n) 



I / -iN m(m — 1) , 
+ I n[n — Ij + mn H 1 v[k, m,n + 1) 



ii k + m + n = 2. 



bNv{k, m, n) 



(11) 

Using that [u{Zt,5,5'))t^Q is a bounded martingale if Z has infinitesimal generator L^'^ 
(Equation ([9|), we obtain the 

Proposition 3.3. u{.,5,5') satisfies: 

{L^'^'u{.,6,6')){k,m,n) = y{k,m,n)\N = k + m + n^2 
u{{0,0,n),6,6') = 1 Vn^2 (12) 
ulik, 0,0), 6,6') = V/c^2 

Our main result in this section is the following theorem studying in detail the deviation of 
u from the neutral case. 

Theorem 3.4. For all {k,m,n) in N^^, the function {5,5') i— )• u{{k,m,n), 5, 5') is an 
analytic function of {6,6') in the neighborhood o/(0,0). Moreover, 



u{{k, m, n),6, 6') = p{k, m, n) — 5v{k, m, n) — 6'v'{k, m, n) + o{\5\ + \5'\), 



where 



v{k, m, n) 



{k — n) 



nY 



m 



{k - nf 



Ar2 



N 



-VN 



v'{k,m,n) = '-^XN + mx'^ + Y{2N -Y)[^ 



Y 



N 2N^ 



VN 



(13) 
(14) 



The sequences x^, i/n, x'j^, and y'j^ are defined as the unique bounded solutions of 2- order 
recurrence equations (Propositions ^^ and U7!\) . 

The proof of this theorem is decomposed in several parts: the existence and formula of the 
two partial derivatives is obtained in Sections 13.3.21 and 13.3.31 and the analyticity of u is 
in Section 14.51 In the following subsections, we consider separately the cases where 6 = 
and 5' = 0. 
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3.3.2 The dependence of u in 5 

To simplify notations, we define: u{{k, m, n),6) = u{{k, m, n), 5, 0). We will show that the 
derivative of n at = is the unique sub-polynomial (i.e. lower than a polynomial function 
in N = k + m + n) solution of a nonlinear recurrenc e equation in (k,m,n). Such result has 
been obtained in IChampagnat and LambertI (120071 ) for the haploid case. Here, the nonlin- 
earity due to both competition and diploid segregation terms generates new mathematica l 
difficulties. We will use some arguments developped in IChampagnat and LambertI ()2007l ) 
and will here focus on the difficulties brought by diploidy. We say that a function / on 
is sublinear if there exists a constant C such that \f{k,m,n)\ ^ C{k + m + n) for every 
(k, m, n). 

Proposition 3.5. For all {k,m,n) in N^^, 

u{{k, m, n), .) is differentiable at 0. Its derivative v{k, m, n) is the unique sublinear solution 
of the system of equations 



{Lv){k,m,n) - 
v{2,0,0) = v{0,0,2) -- 



m{n—k) 

: 



\/{k,m,n) e 



(15) 



Proof. As in the simplest case of haploid populations, we introduce paths of Z, i.e. the 
sequence of states visited by this process. Indeed the fixation probability of the mutant 
allele a if the population Z starts from state (fc, m, n) can be written as the sum of the 
probabilities of every path starting from (A;, m, n) and reaching a state (0, 0, n') with n' ^ 2. 
We then denote by Sf^j^^^^^-^^Q the set of paths linking {k, m,n) ^ F to Q without reaching 
r before and (ii,i2, a path, ij being the j'-th state of the path. We ffiially denote 

by vr^,^ ,^^ the transition probability from state ij to state ij+i for Z. Then 



u{{k, m, re), 6) 



(il,..i;)eS'(fc_„_„)_>ra 



TT • • TT • 



Now vrf ■ is a differentiable function of 5 and the absolute value of its derivative at 
(5 = is bounded independently of (k,m,n) by a constant denoted by Ci. To prove this 
latter assertion, we consider separately the different possible transitions for the population 
in state {k,m,n). For instance the transition probability from state {k,m,n) to state 
(/c + 1, m., re) is 



TT 



{k,m,n),{k+l,m,n) 



h{k{k - 1) + km + m{m - l)/4) 
{N - l){bN + dN + 5m + cN{N - 1)) ' 



Then vrf, \ n , ^ n is differentiable with respect to 5 at 0, and: 



{k,m,n){k+l,m,n) 



86 



5=0 



mb{k{k — 1) + km + m{m — l)/4) 
{N - l){bN + dN + cN{N - 1))^ 



m 



bN + dN + cN{N - 1) b + d + 2c 

Similar computations are made for other possible transitions. Then ^ is differentiable 
with respect to 5 at 5 = and 
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du{{k, m, n),6) 



05 



Then, 



5=0 



(ji,..iOe l'=i 



dirf . 



5<5 







E 



TT ■ ■ TT ■ 



Z'^Sl (fc',m',n')GN3 (ji,...,j;,)G 



5, 



(fc,m,n)-^(fc' ,m' ,ri') 



2^ 2^ ^im- 

eGN3,|le|l = l r>0,(ji,...,i,„)G 

'5'{fc',m',n') + e^ra 



TT 







^ E 

eeN3,||e|| = l 

^ 6Ci IP(A;,m,n)('7r' > the latter sum being lower than 6. 

I'^i 

= 6CiE(fc^^^„)(7r - !)• 

From Proposition 13. H K[k,m,n){'TT) < C2{k + m + n) for a constant C2, which gives that 
u{{k,m,n), .) is differentiable with respect to 5 and that its derivative at v{k,m,n) is 
sublinear. 

Now, identifying the first order terms in 6 in (fT2]) . we see that v satisfies for all (A;, m, n) € 



{Lv){k, m, n) = ^''llf^l) V(A;, m, n) £ N^^ 
v{k,0,0) = v{0,0,n) = if A: ^ 2 and n ^ 2 



(16) 



It remains to prove that the system of Equations (llSp admits a unique sub-polynomial 
solution. Let hhe a sub-polynomial solution of the equation Lh = such that h{2, 0, 0) = 
/i(0,0, 2) = 0. Then {h(Zi))i(^fi is a J"/ -martingale. On Fyi, Lh{k,m,n) = gives 

+ 1, 0, 0) - h{k, 0, 0)) = (d/c + ck{k - l))(/i(A;, 0, 0) - h{k - 1, 0, 0)) V/c ^ 3 

which implies that h = on since /i is sub-polynomial and h{2, 0, 0) = 0. Similarly, 
h = on Tq. Besides, there exists a positive integer such that 

supEfc^m_„(|/i(Zj)p) ^ supEfc^m^„(C|A;t -Fmj -hntl^''). 



Moreover, from Proposition 12. 2( sup^ E^^m.nd^t + mt + ^^tp'^) < +00 for all {k,m,n) in 
N^^. Then the martingale {h(Zi))i^fq is uniformly integrable. From Doob's stopping time 
theorem applied in the stopping time 7r, 

we then liave — ^ 

□ 



Let us now state the following proposition whose proof will be the aim of Section |4l 
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Proposition 3.6. For all {k, m, n) such that k + m + n ^ 2, 



v{k, m, n) = [k — n) 



m N"^ — {k — nf 

+ ^2 VN 



where the sequence of vectors {Zn)i\i>3 = is the unique subpolynomial solution 

of the following system of equations: 

Bnzn+i = Cnzn + Dnzn-i + fN for all N ^ i (17) 
B3Z4 = Cszs + h, (18) 



with 



h / 1 2Af^+4Af-3 

D ._ ( ^ N+1 

^ ■ 2(iV-l)(iV + l) I 2iV2_3 -3 



N+l 



Cn ■■= {b + d + ciN -1)) 



^ 

1 



„ , ^ ^ r ^ 

03:= , A^o^ / , „ s = 



3- I -{d + 2c) J ^ \Ud + 2c) id + 2c) 



3 

II, —I— f ,1 ; V — 11/ II 

Dn 



d + c{N-l)( ^ 



N-1 V N 



N-l 



2Af(Af-l) 

Note here that u(/c, m, n) = —v{n, m, k) and that the comparison between the proportions 
of genotypes AA and aa play a particular role in the value and sign of v. 

3.3.3 The dependence of u in 5' 

For this section we set 6 = 0, i.e. a is a recessive allele, and deleterious when 6' > 0. As 
in the previous section (Proposition 13. 5p li^'^ n '• ^' differentiable and v' is the 

unique sublinear solution of the system 

f Lv'{k,m,n) = 2n{n-i) ^ V(A;, m, n)|/c + m + n ^ 2 

\ v' {2,0,0) = v' {0,0,2) =0 ^ ' 

where y = 2A; + m is the number of A alleles in the population {k, m, n). 

The following proposition (proved in Subsection 14. 4p gives a formula for v' {k,m,n): 

Proposition 3.7. 

v'{k, m, n) := + mx'^ + Y{2N -Y)(^^- ^Vn^ (20) 

/ 2;' 

where xn and yj\f are defined in Proposition \3.6\ and the sequence of vectors z'j^ = I 

\ Vn 

is the unique subpolynomial solution of the following system of equations: 

B'nz'n^^ = C'^z'j^ + D'^z'^_^ + f'^ for all N-^^ (21) 

B'2Z^ = C'2z'2 + f'2, (22) 
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with 



N 



N -I 



2iv2 -2N-1 
1 

2 



N^+N-3/2 I 5 
iV+1 / 



1 3 

Q 13 



{bN + dN + cN{N - 1)) 



2 
2 3 





N 



N •- 



-(d + c(iV-l)) 
/ ^(2iV-l) 



2A^-2 


VN+l 



N-1 
N-2 
N-1 



VN-l 



N 



2N-^ + 3iV^ 



(d + c(iV-l))(4Ar + 2)2(^^ 

VN+l 



m-- 



2 J 2{N + iy 

{bN + dN + cN{N - 1)){2N - 1) 



VN 



+ {d + c{N - 1)){2N' -7N + 



2iV2 

2 ^7Ar , o^ VN-l 



2(A^-i)2 y 



a;2 ~ y2 



X3 + fys 

19 9 

f ys - i2/2 



We now prove Propositions 13.61 and 13. 71 In both cases, the proof is shared in two parts: we 
first prove the result when the fecundity b is small enough compared to the competition 
parameter c, and then we generalize the result to all possible demographic parameters b, 
d, and c. 



4 Proofs of Propositions 13.61 and 13.71 

4.1 Proof of Proposition 13.61 for small b 

To begin with, straightforward calculations give the following lemma: 

Lemma 4.1. (i) If (|13p is true, then v satisfies (|15p if and only if {zn)n'^3 satisfies (|17p . 



([18)) and X2 + |?/2 = ^a^^s + 2?/3- 
{a) (^(fc, m, n))(^ „)gp^3^ is sublinear if and only if {zn)n^s is bounded. 

Notice that Z2 can not be computed; indeed ?;(1,1,0) = — 1;(0, 1, 1) = ^X2 + |y2 and 
v{k,m,n) = elsewhere. 

We then only have to prove that there exists a bounded solution {zn)n^3 to the system of 
Equations (|17p and (jlSp . Notice that if z^ is fixed then for all A^, z^v is fixed, recursively. 
Finding a bounded solution of this system of equations is then equivalent to finding an 
initial condition z (necessarily unique by Proposition 13. 5p such that if zs = z then {zn)n^3 
is bounded. 
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4.1.1 The one-order recurrence relationship satisfied by (zn) 



N 



We change the two-order recurrence system of Equations (|17p and (jlSp into a one-order 
recurrence relationship, so that we can express easily zat as a function of and conversely. 
We easily find that zn satisfies the following recurrence relationship: 



N 

Bnzn+1 = {Cn + Kn)zn + J^(-1)^-'^^(A:, N)h for all iV ^ 3. (23) 

fc=3 

More precisely (|23p is satisfied for A'^ = 3 if = C^^ — C^ and -E(3, 3) = l2- Moreover, if it 
is true for a given N ^ "i then it is true for A'^-l-l as long as i^Af+i = Dn^i{Cn+Kn)~^ B^j 
E{N + 1,N + 1) = h and E{k,N + 1) = Dn+i{Cn + KN)-^E{k, N) for all k G [3,iV]. 
Then the recurrence relationship (I23p is satisfied for every as soon as we can define two 
sequences of matrices {K]\i)i\f^^ and {En)^^^ such that: 

f Kn = Dn{Cn^i + Kn-i)-^Bn-i ViV ^ 4 

Kz = Cz — C3 

E{k, N) = Dn{Cn-i + KN-i)-^E{k, N-l) VA: G [3, iV - 1] 

^ E{k, k)=l2 yk^3 

We then have to prove recursively that F/v := K]^ + Cn is invertible for all N ^ 3. We 
first prove it when c is large enough compared to b. 

4.1.2 Proof of the invertibility of Kn + Cn 

Let us define 

Vn ■■= 



Jj 

1 

Then Fn = {b + d + c{N - 1))Vn + Kn- We now define the matrix Gn ■= Vn + \Kn. 
Then 

Fn = {d + c{N -1))Vn + bCN 

= id + ciN - 1))Vn (/. + ,^,'^_,/ n'Gn) . 

Using the matricial norm ||M|| = supj^j^ 2}(l-^i,i I + l-^j,2|)5 note that HV^"*^!! = N. 

Lemma 4.2. If b ^ then Fn is invertible and ||Giv|| ^ 9 for all N ^ 4. 

This result will be generalized in Subsection 14.21 to all possible parameters 6, d, and c. 

Proof, (of Lemma 14. 2p We prove it recursively. For = 4, we can compute the norm of 
G4. Indeed we have: 



G4 = V4 -I- -DiC^ ^Bs, 



which gives us: 
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/ d+Zc 
' 48(f<+d+2c) 



1 



d+3c 



16{6+d+2c) 

10(d+3c) (d+2c)(d+3c) 

V 16(6+ i^^) 8(fe+d+2c)(6+^) 



1 _ 9(d+3c) \ 
4 64(fe+(i+2c) ' 



27 (d+3c)(b+2(d+2c)+^) 
'64 (f,+d+2c)(;)+i^) 
d+3c 



+ 



32(fe+fil2£) y 



So 



„^ „ fd + Sc/l 1 9\ d + 3c/ 1 30 3 1 

(i + 3c212 212 3 
= ! < < 9 

d + 2c 64 ^ 64 2 ^ 

For all the invertibility of the matrix F/v is a consequence of ||GAr|| ^ 9. Indeed, if 
IIGatII ^ 9, then as long as 6 < 



bV^^GN 



d + c{N -I) 



9bN 



d + c{N -1] 



< 1. 



bVZ^G 



In this case, I2 + ^^^jy_^^ is invertible, and so is Fj\[. Now let us assume that ||GAr|| ^ 9 
for a given N ^ 4 and let us prove that ^ 9. If ||GAr|| ^ 9, then F]\f is invertible 

and we can write Gat+i = Vtv+i + i^iV+i-^jv Hence 



Gn+i — Vn^i + 
Moreover, as long as b ^ 

h + 



D 



N+l ^2 



h + 



hV^^GN 



-1 

N 



B 



N 



d + c{N -1) d + c{N -1) b ' 



bV^^GN 



d + c{N -1) 



bV-'GN 



d+c(N~l) 



9bN 



^ 2. 



d+c{N~l) 



Finally, for all N ^ A, \\Dn+i\\ ^ d + cN and ||V»^^SAr|| ^36 which implies 



||Giv+i|| ^1 + 61 + 



d + 3c 



< 9. 



□ 



As long as 6 ^ c/24. Equation (|23p is satisfied, which allows us to express easily zn as a 
function of 2:3 for all ^ 3. We now prove that there exists a real number z such that if 
z^ = z then (-Zat) is bounded. 
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4.1.3 Boundedness of z 

Let us assume here that 6 < c/24, so that we can use the previous results. Setting 

TV 

:= B~^{C^ + Kjv), and := ^{-lf-^Bj,'E{k, N)h, 

k=3 

we get 

AT / ^ \ 

ZN+i = MnMn-1:.M3{z3 + ^ M^\.M^^gi) = + ^ Pi-^gi (24) 

1=3 \ 1=3 J 

if Pat = Mf^fMN-i-.-M^. To obtain the behaviour of (z^), we then study P/v and g^- 
Lemma 4.3. ||M^"^|| ^ if N is large enough. 

Proof, (of Lemma l4.3p We previously proved (Lemme 14. 2 p that for all A'^ ^ 3, UGatH ^9, 
with Gn = Vn + Then for all iV ^ 3, \\Kn\\ ^ 106. So if 6 < ^, we have 

||i^iv||<c/2 (25) 

for all N ^ 3. Besides, the equation i^Ar+i = D]\f^i(Cj\f + Kn)~^Bn can be detailed, and 
using Equation (|25p . we obtain that 



(26) 



Next, 

iV2 / ^ 

{d + cN){N -2){N -I) V -{N - 1) 
We deduce from this that 

□ 



Notice that if N is large enough 

462 



(28) 



Besides, we have the following lemma for {gN)N- 
Lemma 4.4. g satisfies 

niKT = n -\- , „ , 

N \N 



9N = C + ^ + o(l] (29) 
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Proof, (of lemma From ■= Ef=3(-l)^"''^w^^(^. we deduce 

gN+i = -B~^\iKN+igN + B^\ifN+i (30) 

Moreover, 



>-i 1 2{N-l){N + lf / ^ 

^ 63 + (2iV2 + 4iV-3)(2iV2_3) I 2iv2_3 _i 



and Equation (I26p yields 



Equations (|30p and (|3ip and (|32p give us the result. □ 
Finally, we get interested in ^Pf^gi- Let us recall that Pi = MiMi^i...M^. 

N N 

T.\\pr'9i\\^E\\pr'\\\M- 

1=3 1=3 

From (|29p and Lemma 14.31 {gi)i^3 is bounded and there exists a constant C2 such that 
||M^^|[ ^ ^ when N is large enough. Then X]i=3 B, gi converges and we define its limit 



(33) 



00 

z 

1=3 

The quantity z will be the initial condition, we need to obtain a bounded solution to (iT7|) 
and (|18p as is proved now: 

Lemma 4.5. The sequence (2jv)jV3:3 satisfying (|17p and (|18p . and suc/i i/iai 2:3 = — ^ 
where z has been defined in (|33p J, zs bounded. 

Proof From (|24]) 

(00 \ 00 

/=Ar+l / l=N+l 



00 



(34) 

= -M^^,<77V+i - M^^iM^^2 iM^l^..M^_\){Mi-_\M^')gi. 

l=N+2 

By Lemmas 14.41 and 14.31 and Equation (l28|) . if is large enough, there exists a constant 
C independent from b such that 

M^C^. (35) 

□ 

Proposition 13.61 is now proved for small b. In the next subsection we generalize this result 
to any b. 
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4.2 Generalization to all possible values of b 

Theorem 4.6. For all {k,m,n) such that k + m + n ^ 2, v{k,m,n) is an analytic function 
ofb on 

Corollary 4.7. For all demographic parameters b > 0, d, and c > 0, Proposition \3.6\ is 
true. 

Proof, (of Corollary 14.71 ) From the end of Section I4.1.3| there exists a constant K > 
such that if 6 < Kc, (|13p is true, which gives 



VN 
XN 



A{N -2){N -1) 
N 



viN- 1,0,1), 



N -1 

As long as b < Kc we then have 

m{k — n) 



2N -1 

v{N - 1, 1, 0) - — — — —v{N - 1, 0, 1) 



v{k, m, n) 



N -1 



4(iV - 2) 



v{N - 1, 1, 0) - ^^-lt;(iV - 1, 0, 1) 



+ (k- n)4^ ''^\ v(N - 1,0, 1). 



(36) 



Now from Theorem 14.61 for all {k,m,n) in N^^, v{k,m,n) is an analytic function of b on 
The equality (|36p of two analytic functions on ]0, Erc[ extends on □ 

Before proving Theorem 14. 6( we prove 

Lemma 4.8. For every {k,m,n) in N^^, there exists a strictly positive real number p such 
that Ek,m,n{{l+ pV^) < oo. 

Proof. We define the random number L € N of return of Z in {A^ = 2} before reaching F, 
and 7^^*^ the i-th time of return of Z in {N = 2} {T2^'^ = and T-j^^ = 7(2}) ■ 



Efe,„^,n((l + pf"-) ^ X]^'^'.-." ((^ + p)^'^'^1l=i) as rrlL=i ^ T^'^^^l. 



-(/+!)- 



1=0 

oo 



1=0 (fc',m',n')^r| 
k'+m'+n'=2 or 
(k' ,m' ,n')=(k,m,,n) 



E 



k,m,n 



^(l+p)2 lz^^^^=(k\m',n')'^L=lj 



^ max Efc,,„,,„, ((1 + pfm) X VEfc,^,„ ((1 + pf^^L^i) , 

(k' ,m' ,n')iT\ ^ ' 

k'+m'+n'=2 or '"^ 
(fc',m',n')=(fe,m,n) 



by strong Markov property in 7^'''^ We now define 
S = 



max E.k'^m',n' ((1 + 

(fc ,m ,n )| 



A:'+m'+n'=2 or 
(k' ,m' ,n')={k,m,n) 
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and prove that for every I, 



The result is obviously true for / = and is proved recursively for every / by using strong 
Markov property in Tg' as previously. Now from Proposition 12. H for every {k, m, n) 
there exists p > such that IEA:,m,n((l + ) < c«. Then by the Dominated Convergence 
Theorem, Efcm„((l + p)'^^^ilL>i) — > F^mniL ^ 1) < 1. Hence there exists po such that 

if p < Po, S < 1 and then „i,n((l + p)^) < oo. □ 

Proof, (of Theorem 14. 6p We need to study the dependence of the probability u in the 
fecundity parameter b, so we denote by u{{k, m, n), 6, b) the fixation probability of allele a 
when Zq = (k,m,n) and v{{k,m,n),b) its derivative with respect to 6. If u{{k,m,n), ., .) 
is an analytic function of (6, 6) on M"*"* x M, then v{{k, m, n), .) is an analytic function of b 
on R+*. Now, 

u{{k,m,n),6,b) Yl 4fi2-4tiii^ 

where T^^'^i^^^ is the transition probability from state ik to state ik+i and an analytic 
function of (6, 5) on IR"'"* x R. u is then the simple limit of analytic functions on M"*"* x M. 
By (9.13.1) and (9.13.2) of lPieudonni (ll969l V a sequence of analytic functions {fn)n defined 



on an open set S" of C which converges simply towards a function f on S , is proved to 
converge uniformly on every compact subset of S as long as {fmn G N} is relatively 
compact. We extend the functions T^ifii^^-^ on the open set x E2 where /3 € M"*"* and 

= {z£ C\Re{z) > 0, \Im{z)\ < l3Re{z)}, 



E'. 



{z G C||i?e(z)| < d/2, \Im{z)\ < p{d - \Re{z)\ + 2c)}. 



We set b = br + ibi € £'f , 5 = 5r + i5i ^ E^ and denote by P^k^rn,n){k',m',n') analytic 



extension of T^^^fj^ ^/ on E^ x ■ For all (6, (5) G E'f x E2 and for all (k, m, n) and 
{k',m',n') neighbors in N^: 



->b,S 



< _|_ «2 p6r,<5r 

^ V i -r ^ ^(^k,m,n)(k',m',n') 



{k,m,n){k' ,m' ,n') 

Indeed, let us make the computation if {k',m',n') = (k,m — l,n). 



./1 _|_ R2 

V ^ ^ r " {k,m,n){k' ,m' ,n') ' 
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p: 



b,5 



{k,m,n)(k,m—l,n) 



{d + 6 + c{N - l))m 



bN + dN + 6m + cN{N - 1) 

\{d + 6 + c{N - l))m\ 
Re{bN + dN + 5m + cN{N - 1)) 

{d + 6r + c{N - l))2m2 + (52m2 
brN + dN + 5rm + cN{N - 1) 



{d + 6r + c{N - l))m \\ 1 + 



id+5r+c(N-l)y^ 



brN + dN + 6rm + cN{N - 1) 



brN + dN + 5rm + cN{N - 1) ^ 

— V -L ^ /J -^(fc,m,n)(fc,m-l,n) 



Computations are similar for other possible transitions. Then since y^l + /32 ^ i -)_ 



E E 

1^1 {«i,--«i)GS(fe_^,„)_>r<j 



L 



TT ■ ■ ...IT- 

n«2 H~iH 



since, if Tr^ < co, Tr^ = 7r- □ 
In the following subsection, we establish some properties of the derivative v{k,m,n). 
4.3 Boundedness and sign of v 

Proposition 4.9. (i) For all demographic parameters b, d and c, v is a bounded function 
of (k, m, n). 

(ii) Vk,m,n = ^(k,m,n) Jq Lv{Zt)dt ^ whcrc T = inf{t, h = rif or rrit = nt = 0}. 

[iii) v{k,m,n) has the same sign than k — n. 

Proof, (i) is a consequence of Equation (|35p and (iii) is a consequence of (ii). For (ii), by 
Proposition 13. 6[ it suffices to prove the result when k > n. The function v being bounded 
in {k,m,n) (by (i)), Dynkin's formula stopped at the stopping time T gives us that 



T 



Lv{Zt)dt 



Using that v{Zt) = (from Proposition 13. 6p . we get the result. 



□ 
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Notice that the sign of 6 is not sufficient to know whether the allele a has a larger fixation 
probability than a neutral allele, or not. This property depends on the initial genetic 
repartition of the population: if there are more alleles A (resp. a) initially, then allele a 
has a lower fixation probability than a neutral allele if and only if (5 > (resp. 5 < 0). 
In Section \5\ we will get interested in the particular case where the allele a is a mutant 
appearing in the population. In this case, at mutation time, there is only one individual 
with genotype Aa and no individual with genotype aa, then the population starts from a 
state of the form (k, 1, 0). The fixation probability of allele a is then: 



//, n^ ^^ ^ .f^ k{2k + 1) ^ . ^. 



4.4 Proof of Proposition 13.71 

As in computations for v, Proposition 13 . 7l is true if we can find a bounded sequence (z'j^) 
which is solution of (l2T]) and (122p . To prove this, we use a similar proof as for 5' = (Section 
O]) . Setting 

hk = fk yk^4. 
^3 = /a - D'^C!f^f2, 
we easily obtain that for all N ^ 3: 



N 

B'nz'n+1 = (C'r, + K'M + Y.{-1)'E'{N, k)hk 

fc=3 



(37) 



with 



K'n = D'^iC'N^i + K-i)-'B'^_, ViV ^ 4 
E'{k,k)=l2 V/c ^ 3 

E'{N, k) = D'^{C'^_^ + K'^_^)-^E'{N - 1, k) 

= K'j^B'^\E'{N -l,k) ViV ^ A: + 1 

Notice here that the detailed computation of shows that /13 does not depend on X2 
and y2 (which are not known) but only on X2 + |2/2- The only difficulty in adapting the 
proof of Section 14.11 is when proving that there exists a constant C such that for all A^, 
||-B^^/iAr|| ^ Note that we have 

1 A^-1 N + 1 



\ 2 

From Equations (j27|, (|29|) and ([Ml), 



b (2A2-2A-l)(iV2 + ^_3/2) + i/2 
jV^+jV-3/2 1 \ 

N+l N+l 1 

1 -2N -I ' 



Ci C2 ^( I 
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Then 



N+1 

2iv2 -2N -1 



IN 



0(1) and 



O 



We now know that if the birth parameter b is small enough compared to c, then v' is 
effectively defined as in Formula (|14p . To generalize this result to all possible values of 
parameters h and c, we adapt the proof of Theorem 14.61 and Corollary 14.71 to (5', without 
any difficulty. Note here that for all demographic parameters, v' is a positive bounded 
function of {k,m,n). 



4.5 Proof of the analyticity of u{k,m,n) 

To conclude these results, we now prove that u{{k,m,n),6,6') is an analytic function of 
{6,6') in the neighborhood of (0,0). 

Proof. We use analytic extension arguments as in the proof of Theorem 14.61 Here 6 
and 5' are complex numbers, denoted hy 6 = 6r + iSi and 6' = 5'^. + i6[. We take 
{5,5') G with = {z e C||i?e(/)| < d/2,\Im{z)\ < p{d - \Re{z)\ + 2c)}, and 

denote by T^f^^ ^/ the transition probability for Z from {k, m, n) to one of its 



neighbor {k',m',n') and P^!^ , ,^ the analytic continuation of 7r"r s/,, , ^ on 

(^^)2. Then, 



5,5' 



^5,5' 
{k,m,n)(k' ,m' ,n') 



< n _|_ fj'2\pSr,5'r 

55 1^1 -r ^ J-^(k,m,n){k',Tn',n') 

- Cl _L /^SW-^-'-^r 

— l-L-l-p )'^(^k^rn,n){k',m',n')- 



Indeed, it is proved by making the computation for all possible transitions as in the proof 
of Theorem 14.61 and the conclusion follows similarly. □ 



Theorem 13.41 is now proved. 



5 Mutational scale: convergence and extinction vortex 



Understanding and quantifying the extinction risk of a popula tion is a very iniportan t 



issue, in particular within the framework of species conservation iGilpin and Soule 



We now get interested in a phenomenon called "mutational meltdown" iLvnch et al. 



1986 ) 



19951): 



within small populations, inbreeding favors the fixation of delet e rious alleles that would dis- 
appea r in an infinit e size p opulation Crow and Kimura ( 1970l ): Champagnat and Meleard 
(j201ll ): iMetz et al.l (Il996[ ). This phenomenon is then characterized by more and more 
frequent fixations of deleterious all eles, w h ich cr e ates an extinction vortex and leads to a 
rapid extinction of the population iLandd (jl994[ ) ; iGilpin and Soulel (119861 ) . We wish now 
to observe this acceleration of mutation fixations. To this end, we introduce mutations in 
our model, and consider a different time scale. 
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5.1 General model 



As introduced in Section |2l each individual is now characterized by its genotype x € G := 
{{A,C,Q,T}^}^- Now every DNA strand can now mutate during the individual lifetime, 
at rate fj,K := fi/K. X is a scaling parameter that will go to infi n ity, fo l lowing a rare 
mutat ion hypothesis, which is usual in evolutionary genetics Lande ( 1994 ): Champagnat 
(j2006l ). For every a,a' € {A,C,g,T}'^, we define the probability M{a,a') that a DNA 



strand a mutates to a' knowing that a mutates. The population can then be represented 
at time t by 

i=l 

where Nf- is the size of population at time t and xl'^ is the genotype of the i-th 
individual in population Z^ at time t. Z^ belongs to the discrete space: 

( N \ 



where E is equipped with its discrete topology and the norm r{fi, v) = X^xeG 
We denote by B([0, oo), E') the Skhorohod space of left limited right continuous functions 
from to endowed with the Skhorohod topology. We denote by 6(x, Z) the birth 
rate of an individual with genotype x in the population Z, and assume that there exists 
a constant C such that for every Z with size iV, ^ h{x, Z) ^ CN. As in Section [21 

x-eG 

individuals can die either naturally, or due to competition with other individuals, and when 
the population size reaches 2 we assume that no death can occur. We denote by d{x, Z) 
the death rate of a given individual with genotype x in the population Z and assume that 
for every x, d{x, Z) is bounded below by some positive power of the population size. For 
all > and for all real bounded mesurable function f on E, if Z = X^^^ with 
^{t) — (^x^\x2^), the generator of the Markov process is: 



= Kx, Z){f{Z + 4) - f{Z)) 
xeG 

N 

+ Y,d{x,,Z){f{Z-5^,.,)-f{Z)) 



i=l 
N 

+ E ^ E M{xf,y)U{Z - 5,,, + <5(^^^„p - f{Z)) 
i=i j/e{Ac,g,r}G 

N 

+ Ef E M{xf,y){f{Z-5,,.,+5^^^,J-f{Z)). 

i=l y£{A,C,g,T}G 

Notations: When the population is monomorphic, i.e. every individual has same genotype 
X, we assume that the population follows a neutral logistic birth- an d-death process as 
presented in Section 13. 2| and we denote by b{x), d{x) and c{x) the birth, and natural 
and competition death rates (denoted 6, d, and c in Section 13. 2p . For all demographic 
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parameters b, d, and c, we also define the stationary law l{.,b,d,c) of the population size 
of this neutral logistic birth-and-death process. / satisfies the stationary equations system: 



' 6(iV - l)/(iV - 1, b, d, c) + {d + cN){N + 1)/(A^ + 1, b, d, c) 

= N{b + d + c{N -l))l{N,b,d,c) ViV^3 

26/(2, b, d, c) = 3{d + 2c)l{3, b, d, c). 
Then for all iV ^ 2, 

N ^} d + kc 

mAd,c):= ^ '=1^ . (38) 

We now rescale time when K goes to infinity, in order to observe mutation apparitions. 
More precisely, the mean time of apparition of a mutation being equal to ~ we 

accelerate time by multiplying t by K. 

5.2 Convergence and limiting process in the adaptive dynamics asymp- 
totics 

Theorem 5.1. For all < ti < ... < the n-tuple (Z^^^, Z^^^) converges in law 
towards the process [Nt-^^Sst-^, ^tn^Stn) where 

(i) {St)t>o is a Markov jump process that jumps from a homozygous genotype x^^^ = 
{xi,xi) to another homozygous genotype x*-^-* = (x2,X2) where xi and X2 are in 
{A,C,Q,T]^, at rate r(x(^), x^^)). 

(ii) 

r(x(^),x(2)) = 2/iM(xi,X2) 

X ^ iV/((iV- 1,1,0), xW,x(2))/(A^,6(x«),d(x«),c(x«)), ^ ' 

N=2 



where f{{k,m,n),x^^\x^'^^) is the probability that, starting from k individuals with 
genotype x^^\ m with genotype {xi,X2), andn with genotype x^'^\ the population gets 
finally monomorphic with genotype x^^) . In the particular case where only the natural 
death rate differs between individuals with genotypes x*-^^ and x^'^\ as in Equation 

f{{N - 1, 1, 0), xW , x(2)) = u{{N - 1, 1, 0), d(xi, X2) - d(xW), d(x(2)) - d(xW)) 

where d{x^^^), d{{xi,X2)), and d{x^'^^) are the respective natural death rates of indi- 
viduals with genotype x^^\ {xi,X2) and x^"^^ (the generalization of genotypes AA, Aa, 
and aa in Section \3.2\) . and u has been studied in Section [3 
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(iii) Conditionnally to {St^^, St, 



{x 



(1) 



..,x 



'^"•'), the random variables Ni 



are mutually independent and for all i, Nf- has law l{.,b{x^'^^] 



ti, ■■■ 

d(x«),c(x(^))). 



At this mutational time scale, the process {Nt6st)t^o describes the successive fixations 
of mutations. Indeed, a jump of the limiting process S corresponds to a change in the 
genotype of every individual of th e population, i.e. a mutati o n fixa tion. This previous 



theorem is directly obtained from IChampagnat and LambertI (120071 ). except from a few 
details in the proof, which are given in Appendix lAl 



5.3 The extinction vortex 

In this section we focus on the jump process S and assume that all mutations have the 
same effect than described in Equation (llOp . i.e. when xi mutates to X2, individuals with 
genotypes (xi,X2) and x^^^ all have same fecundity b and competition parameter c, 

but 

(i(xi,X2) = d(xW) + 5, and d(x(2)) = + 5'. 

What is more, we exclude overdominance cases by assuming that 5 < 6' . We denote by 

oo 

T{d,6,5')= Nu{{N - 1,1,0), d,6,5')l{N,d) (40) 

N=2 

the jump rate of the limiting process S of Theorem 15.11 (Equation (j39|) ) when individuals 
have birth rate b, natural death rate d, and competition rate c (the dependence in param- 
eters b and c is hidden, to simplify notations, we assumed fj, = 1/2). This rate is also the 
rate of fixation of a deleterious mutation with size {6, 6'). Let us recall that the extinction 
vortex is due to more and more rapid fixations of deleterious mutations in the population. 
We then wish to prove that the mean time to fixation of a deleterious mutation decreases 
when the number of already fixed mutations increases. Now when a deleterious mutation 
gets fixed, the natural death rate of all individuals is increased by 6' . The vortex is then 
due to the fact that the mean time to fixation of a deleterious mutation is a decreasing 
function of the natural death rate d of individuals, which is proved in the next theorem. 

Theorem 5.2. If 5 > and 5' > 6, and if b is small enough, the mean time to a jump of 
process S T{b,d,c,5,5') = 1/T{b,d,c,5,5') is a decreasing function of d, the natural death 
rate of individuals. 

Here we underline the dependence of all quantities in d, by denoting respectively by 
u{{k,m,n),d,5,5'), v{{k,m,n),d), and v'{{k,m,n),d) the fixation probability defined in 
Section [3] and its derivatives, when individuals have natural death rate d. We also denote 
by l{-,d) the stationary law of the population size (Equation (I38p ). We first need to prove 
the following lemma: 

Lemma 5.3. // d and d' are two non negative real numbers such that d' > d, then there 
exists an integer Nq such that for all N ^ Nq, l{N,d') ^ l{N,d), and for all N > Nq, 
l{N,d') < l{N,d). 

Proof Let us define q{N) = Equation §8^ gives us that q{N + 1) = §^q{N), 

then if d' > d, q{N) is a strictly decreasing function of A^. Next, 
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1 V^CXD 1 TTOO b 

2 Z^i=2 i 11 1=2 



/„s _ ^ ^i-z. % '■'■3=2 d+jc 
"\ I ~ I v^oo 1 T-roo b ' 
2 2-ji=2 i 117=2 d'+jc 



hence q{2) > 1. Finally, if q{N) > 1 for all N then /(iV,d') > /(7V,d) for all N which is 
absurd as l{.,d) and l{.,d') are probability measures. Then there exists an integer A'o such 
that for all N > iVo, q{N) < 1 and for all ^ Nq, q{N) ^1. □ 



Proof. (Theorem 15. 2p From Theorem 13. 4| the mean time to fixation of a mutation is 
T{d,6,6') = 1/T{d,6,6') with 



r{d,6,5') = ^ 



N{6v{{N - 1, 1, 0), d) + 6'v'{(N - 1, 1, 0),d))l{N, d) 



.N=2 



(41) 



where the differentiability of the infinite sum in (|40p is obtained as in the proof of Propo- 
sition 13.51 Then if d' > d, 



T{d', 6, 6') - rid, 5, 6') = ^ N{6v{{N - 1, 1, 0),d) + 6'v'{{N - 1, 1,0), d))l{N, d) 



N=2 

00 



- N{6v{{N - 1, 1, 0),d') + 5'v'{{N - 1, 1, 0),d'))l{N, d') 

N=2 

+ o{\5\ + \5'\) 

CO 

= 6 J2 Nl{N,d){v{{N - 1, 1,0), d) - v{{N - 1,1,0), d')) 



N=2 
00 



-(5^ iVu((iV- 1,1,0), d')(/(A^,(i') d)) 

N=2 

00 

+ 5'YNliN^d){v'{{N-l,l,0),d)-v'{{N-l,l,0),d')) 



N=2 

00 



-S'Y Nv'i{N - 1,1,0), d'){l{N,d') - l{N,d)) 

N=2 

+ o{\5\ + \5'\). 
Defining Nq as in Lemma |5.3| we obtain: 
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T{d',6,5') -T{d,6,6') = SJ2 Nl{N,d){v{{N - l,l,0),d) - v{{N - l,l,0),d')) 

N=2 

CO 

+ S'Y^ Nl{N,d){v'{{N -1,1,0), d) -v'{{N -1,1,0), d')) 

N=2 

oo 

- 6 i^^iN - 1, 1, 0),d') - Nov{{No - 1, 1, 0),d')){l{N, d') - 1{N, d)) 



N=2 

CO 

{Nv'iiN - 1, 1,0), d') - Nov' {{No - 1, 1, 0),d')){l{N, d') - 1{N, d)) 

N=2 

+ o{\5\ + 1^1) the added terms being equal to 0, 

(42) 

which gives, if w{{k, m, n),d) = 5v{{k, m, n),d) + 5'v'{{k, m, n), d), 

CO 

T{d',5,5')-T{d,5,5') = Nl{N,d){w{{N -1,1,0), d) - w{{N - 1,1,0), d')) 

N=2 

°° (43) 
-Yi^MiN -1,1,0), d')-Now{{No- 1,1,0), d')){l{N,d')-l{N,d)) 

N=2 

+ o{\S\ + \5'\) 

Let us now prove first that N i-^ Nw{{N — 1,1,0), d') is increasing and then that d i— )■ 
{w{{N — 1,1,0), d) is decreasing. These two results imply Theorem 15.21 and will be conse- 
quences of the two following lemmas. Notice that the infinitesimal generator L (Equation 
(|lip ) is the sum of two generators 

{Lf){k,m,n) = {Lhf){k,m,n) + {Ldf){k,m,n) 

where 

3 

Li,f{Z) = Y,UZ){f{Z + ei)- f{Z)), and 

2 = 1 

Ldf{Z) = {d + c{N-l)) 

X [kf{k — 1, 771, n) + mf{k, m — l,n) + nf{k, m,n — 1) — Nf{k, m, n)]. 

Since dLw/dd = (from (fT5]) and (IT9])). 

dw{.,d) \ -{Ldw{.,d)){k,m,n) 
^^^J (''"^'"^ = d + c{N-l) ■ ^"'^ 

Notice also that 

{Ldw{.,d)){N -1,1,0) = {d+c{N -1))[{N -l)w{N -2,l,0,d)- Nw{N -l,l,0,d)], (45) 

so if we prove that {Ldw{., d')){N -1, 1,0) ^ for all ^ 2, then N ^ Nw{{N-l,l,0),d') 
is increasing. In fact we prove the 
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Lemma 5.4. If b is small enough and 6' > 5, then for all {k,m,n) in N^^, 

dw{.,d) ~ 



L- 



dd 



{k, m, n) ^ 0. 



Proof. (Lemma 15. 4p There exists a constant C > such that for all (k,m,n) in N^^ 

b 



{Lw{., d)){k, m, n) = {Ldw{.,d)){k, m,n) [ 1 + 



d + c{N - 1) 
b 



{Ldw{.,d)){k,m,n] 



d + c{N -I) 
5m{k — n) + 5'nY {5' — 5)nm + k{6m + 26'n) 



2N{N - 1) 
-C{km + mn + kn) 



2N{N - 1) 



2N{N - 1) 

Next, detailed computations give us that there exists a constant C' such that 

b 



{Lbw{.,d)){k,m,n) 



d + c{N -1' 



■{Ldw{.,d)){k,m,n) 



€ 5b 



\k -n\[ m|xAr+i - XAr_i[ H — lUN+i - VN-i 



+ 6'b[Ym\xN+i - XN-i \ + mA^|x^+i - x'j^_i \ + (2A^ - Y)Y\yN+i - yN-i\ 
+ {2N-Y)Y\y'^_,,-y'^^,\)] 
km + mn + kn 



+ bC' 



N 



'-{\xN+l \ + \xN~l \ + \x'n^i \ + \x'm_i\ 

+ \yN+i \ + \yN-i \ + \y'N+i\ + bjv-il 



Finally, from Equations (|34p and (|35p . when b is small enough, there exists a constant C" 
independent from b such that |3;Ar+i| < and the same result is true for y, x' and y'. 
Then if b is small enough. 



Liyw{k, m, n) 



d + c{N - 1 



-Liiw{k, m, n) 



C (km + mn + kn) , 
< 2N{N-l) ^ik,m,n)enl^ 



which gives that Ldw{k,m,n) ^ for all {k,m,n) and the result by (|44p . 
We finally prove that 

Lemma 5.5. If b is small enough and 6' > 5, then for all {k,m,n) in N^^, 
dw{{k, m, n),d) 



□ 



dd 



-E, 



{k,in,n) 







(46) 



Proof. (Lemma 15. Sp We use Dynkin's formula, stopped at time Tjv = inf{t > 0, A^f ^ N}: 



dw{ZT^f^TN,d) _ dw{Zo,d) 



dd 



dd 



r dw{.,d) 

] {Z,)ds 
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where (M(/^5"^)t>o is a martingale. Since {Ldw/dd{.,d)){k,m,n) ^ for all {k,m,n) 
(Lemma 15. 4p . then ii k + m + n = Nq, 



N^No 



and 

{dw{ZTj,ATN,d)/dd - dwiZo,d)/dd - Mtj,aTn)n;=^No 

are two increasing sequences of positive variables since Tat ^ T/v+i when N ^ A'^o = k + 
m + n. From the monotone convergence theorem, since Tr A T/v — > p.s. (Proposition 



Iff 

'^{k,m,ri) 



{Zs)ds 



and 



^ik,m,n) ^ Mt.aT^ j 



Af-s>oo ' ' ' 



dw{ZT^,d) dwiZo) 



dd dd 

Using dw{ZTj.,d)/dd = Mt^ = 0, we get the result. □ 

Finally, (|44p . (|45p and Lemma 15.41 implv that N i— )• Nw{{N — l,l,0),d) is an increasing 
function of N, and Lemmas 15 . 4 1 and 15 . 5 1 give that 'w{{N — 1, 1, 0),d) is a decreasing function 
of d. □ 

5.4 Numerical results 

Equation (|34p allows us to approximate the sequences {zn)n^2 numerically, and we do 
the same for {z'j^)]\[^2 and then for r (Equation (14ip ). Figure [T] shows the mean time 
T to fixation of a deleterious mutation as a decreasing function of d (Theorem 15. 2p . for 
va rious values o f b, 6, and 5'. For more biological analysis and numerical results, we refer 



to Coron et al.. 



A Proof of Theorem 15.11 



In this article we consider a diploid population and, as seen in Theorem 13.41 the diploidy 
generates interesting formulas for the fixation probability of a non neutral allele. More 
precisely, this fixation probability is a function of the initial genetic repartition in the 
population (parameters k, m, and n) and cannot be reduced to a function of the initial 
numbers of allele A and a in the population, as for a haploid population. At the mutational 
time scale ( Section [5]) , this leads to mutation fixation rates that are different than those 
obtained in IChampagnat and LambertI (120071 ) for the haploid case. 

Howe ver, the proof of Theorem 15.11 can be seen as an extension of the proof of Theorem 



JO 
rtl t 



3.1 of IChampagnat and LambertI (120071 ) . to the cases where mutations occur during life 



29 




Figure 1: |(A)[ Relationship between T, the mean time to fixation of a deleterious mutation, 
and the population intrinsic death rate c? as a function of selection and dominance. Open 
symbols: recessive mutation {5 = 0); closed symbols: additive mutation (5 = S' /2); circles: 
6' = 0.1; diamonds: 6' = 0.2. Other demographic parameters are 6 = 10, c = 0.1, and 
m = 1 |(B)[ Relationship between the mean time to fixation of a deleterious mutation T 
and parameters b and d. Each curve corresponds to a fixed value of b. Other parameters 
are 6 = 0.05, 6' = 0.1, c = 0.1 and m = 1. 



and not at birth, and where no death can occur when there are two individuals in the 
popul ation. We now explain why those differences do not hamper the proof of Theorem 
3.1 of IChampagnat and LambertI (j2007l ). which is constituted of three lemmas. 



First lemma: Lemma 6.2 of IChampagnat and LambertI (120071 ) proves that there are no 
mutation accumulations when parameter K goes to infinity. Using Proposition 12. 2| the 
lemma and its proof remain true in our model. 



Second lemma: The first part of Lemma 6.3 of IChampagnat and LambertI (120071 ) gives 
the limiting law of Kti and of the population size at time ri when K goes to infinity, 
where ti is the first mutation apparition time for the population . Here the proof is 
similar but uses different rates: as long as t < ti, if the population is initially monomorphic 



with genotype x, the population size (A'7^)o<t<ri follows a birth and death process with 
birth rate b{x,i5x)i and death rate d{x,i5x)i when = i, and ti is the first point of an 
inhomogeneous Poisson point process with intensity {2fi/ K)Nf^ . Then for any bounded 
function / : N \ {0, 1} ^ M, 



E(/(iV, 



K 



,K}) = 2/i (\{f{N^,)N^,e-^^'^St^^^^-ds) 
Jo 

= 2^ rE(/(iVL)iVLe-''^/^^o"^°'^^d«) 



since the law of does not depend on K. The ergodic theorem finally gives us that 
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w here is a random variable with law I defined by (|38p . The second part of Lemma 6.3 
of IChampagnat and LambertI ( 20071 ) gives us that supK>i^n5 i^n) ^ Here the proof 
needs to be slightly changed as the population size does not reach 1 in our model. We then 



define Lt = l^^o-. 



:2} 



du and have 



E{NP,+^expi-=^LKs)ds). 



2fi 



We finally prove that there exis t A, A', C > such that P(Lf ^ At) ^ Ce as in 



Champagnat and LambertI (120071 ). by defining Si 
mf{t ^ Si : iV° = 3}. 



inf{s ^ ti^i : iV° = 2} and U 



Third lemma: The third lemma gives the behavior of pi, the first time where the pop- 
ulation becomes monomorphic, and Vi, the genotype of individuals at time pi, if the 
population initially contains 2 genotypes x and y. This lemma and the end of the proof of 
Theorem 15.11 are easily generalized to our model. ■ 
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